Device and method for determining a center of a trailer tow coupler

ABSTRACT

A method for determining a location of a target positioned behind a tow vehicle is provided. The method includes receiving images from a camera positioned on a back portion of the tow vehicle. The images include the target. The method includes applying one or more filter banks to the images. The method also includes determining a region of interest within each image based on the applied filter banks. The region of interest includes the target. The method also includes identifying the target within the region of interest and determining a target location of the target including a location in a real-world coordinate system. The method also includes transmitting instructions to a drive system supported by the vehicle. The instructions cause the tow vehicle to autonomously maneuver towards the location in the real-world coordinate system.

CROSS REFERENCE TO RELATED APPLICATIONS

This U.S. patent application claims priority under 35 U.S.C. § 119(e) toU.S. Provisional Application 62/663,354, filed on Apr. 27, 2018, whichis hereby incorporated by reference in its entirety.

TECHNICAL FIELD

This disclosure relates to a device and method for determining a centerof a trailer tow coupler.

BACKGROUND

Trailers are usually unpowered vehicles that are pulled by a powered towvehicle. A trailer may be a utility trailer, a popup camper, a traveltrailer, livestock trailer, flatbed trailer, enclosed car hauler, andboat trailer, among others. The tow vehicle may be a car, a crossover, atruck, a van, a sports-utility-vehicle (SUV), a recreational vehicle(RV), or any other vehicle configured to attach to the trailer and pullthe trailer. The trailer may be attached to a powered vehicle using atrailer hitch. A receiver hitch mounts on the tow vehicle and connectsto the trailer hitch to form a connection.

In some examples, the trailer includes a coupler tongue (i.e., thetrailer hitch), a tow-bar connecting the coupler tongue to the trailer,a jack support for supporting the trailer before connection to thevehicle that pivots about a wheel axle center of the trailer wheels.Trailer connections and features vary greatly. For example,trailer-tow-bar-coupler features include different shapes, colors,tow-bar types, coupler types, jack support types, tongue attachment to aframe of the trailer, location of the jack support, extra objects thetrailer supports, size and weight capacity of the trailer, and number ofaxles. Therefore, the different combinations of the above featuresresults in many different trailer models, making the trailer types verydiverse. In addition, in some examples, a trailer owner personalizeshis/her trailer, which further makes the trailer more different.Therefore, a vehicle having a trailer detection system configured toidentify a trailer positioned behind the vehicle, may have difficultyidentifying a trailer or a trailer hitch due to the different trailertypes available. Moreover, the vehicle identification system may have ahard time identifying a trailer positioned in the back of the vehiclebecause the trailer may be positioned on a variety of surfaces such asgrass, dirt road, beach, etc. other than well maintained road that makeit difficult to identify. In current trailer identification systems, ifthe identification system has not previously stored such an environment,then the identification system is not able to identify the trailer. Inaddition, current trailer identification system that use a vision-basedapproach to detect the trailer tends to generalize the features of thetrailer features, i.e., the tow-bar, the coupler, which results in afailure to detect a trailer having non-common features, or a customizedtrailer, or personalized trailer in an uncommon environment (i.e., anenvironment different from a well maintained road). In addition, theproduction of the trailer identification system hardware to detect thedifferent trailer is very costly due to the large amount of dataprocessing needed. Therefore, the current trailer detection systems arenot able to distinguish a specific trailer combination in multipletrailer scenarios.

It is desirable to have a system that tackles the afore-mentionedproblems by quickly and easily identify a trailer-tow-bar-couplercombination regardless of the shape and surface the trailer ispositioned on.

SUMMARY

One aspect of the disclosure provides a method for determining alocation of a target positioned behind a tow vehicle. The methodincludes receiving, at data processing hardware, images from a camerapositioned on a back portion of the tow vehicle and in communicationwith the data processing hardware. The images include the target. Themethod also includes applying, by the data processing hardware, one ormore filter banks to the images. The method includes determining, by thedata processing hardware, a region of interest within each image basedon the applied filter banks. The region of interest includes the target.The method also includes identifying, by the data processing hardware,the target within the region of interest. The method also includesdetermining, by the data processing hardware, a target location of thetarget including a location in a real-world coordinate system. Themethod also includes transmitting, from the data processing hardware,instructions to a drive system supported by the vehicle and incommunication with the data processing hardware. The instructions causethe tow vehicle to autonomously maneuver towards the location in thereal-world coordinate system.

Implementations of the disclosure may include one or more of thefollowing optional features. In some implementations, the method furtherincludes tracking, by the data processing hardware, the target while thetow vehicle autonomously maneuvers towards the identified target. Themethod may also include determining, by the data processing hardware, anupdated target location. In some examples, the method includestransmitting, from the data processing hardware, updated instructions tothe drive system. The updated instructions cause the tow vehicle toautonomously maneuver towards the updated target location.

In some implementations, the camera includes a fisheye camera capturingfisheye images. In some examples, the method further includesrectifying, by the data processing hardware, the fisheye images beforeapplying the one or more filter banks.

In some implementations, the method includes receiving, at the dataprocessing hardware, training images stored in hardware memory incommunication with the data processing hardware and determining, by thedata processing hardware, a training region of interest within eachreceived image. The training region of interest includes a target. Themethod may include determining, by the data processing hardware, the oneor more filter banks within each training region of interest. In someexamples, the method further includes: identifying, by the dataprocessing hardware, a center of the target, where the target locationincludes a location of the center of the target.

The target may be a coupler of a tow-bar-coupler supported by a trailer.The images may be a top-down view of the tow-bar-coupler. In someexamples, the target is a trailer positioned behind the tow vehicle andthe target location is a location of a trailer bottom center at atow-bar. The images are a perspective view of the trailer.

Another aspect of the disclosure provides a system for determining alocation of a target positioned behind a tow vehicle. The systemincludes: data processing hardware and memory hardware in communicationwith the data processing hardware. The memory hardware storesinstructions that when executed on the data processing hardware causethe data processing hardware to perform operations. The operationsinclude receiving images from a camera positioned on a back portion ofthe tow vehicle and in communication with the data processing hardware.The images include the target. The operations include applying one ormore filter banks to the images. The operations include determining aregion of interest within each image based on the applied filter banks.The region of interest includes the target. The operations includeidentifying the target within the region of interest and determining atarget location of the target including a location in a real-worldcoordinate system. The operations include transmitting instructions to adrive system supported by the vehicle and in communication with the dataprocessing hardware. The instructions causing the tow vehicle toautonomously maneuver towards the location in the real-world coordinatesystem.

Implementations of this aspect of the disclosure may include one or moreof the following optional features. In some implementations, theoperations further include tracking the target while the tow vehicleautonomously maneuvers towards the identified target. The operations mayinclude determining an updated target location and transmitting updatedinstructions to the drive system. The updated instructions causing thetow vehicle to autonomously maneuver towards the updated targetlocation.

In some implementations, the camera includes a fisheye camera capturingfisheye images. The operations may include rectifying the fisheye imagesbefore applying the one or more filter banks.

In some examples, the operations further include: receiving trainingimages stored in hardware memory in communication with the dataprocessing hardware; and determining a training region of interestwithin each received image. The training region of interest includes atarget. The operations may also include determining the one or morefilter banks within each training region of interest. The operationsfurther include identifying a center of the target, where the targetlocation includes a location of the center of the target.

In some examples, the target is a coupler of a tow-bar-coupler supportedby a trailer. The images may be a top-down view of the tow-bar-coupler.The target may be a trailer positioned behind the tow vehicle and thetarget location is a location of a trailer bottom center at a tow-bar.The images are a perspective view of the trailer.

The details of one or more implementations of the disclosure are setforth in the accompanying drawings and the description below. Otheraspects, features, and advantages will be apparent from the descriptionand drawings, and from the claims.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic top view of an exemplary tow vehicle at a distancefrom a trailer.

FIG. 2 is a schematic view of an exemplary vehicle.

FIG. 3 is a schematic view of an exemplary arrangement of operationsexecuted during a training phase.

FIG. 4A is a schematic view of an exemplary image that includes thetow-bar and coupler.

FIGS. 4B and 4C schematic view of an exemplary image that includes thetrailer at different distances from the tow vehicle.

FIG. 5 is schematic view of an exemplary arrangement of operations fordetermining a region of interest proposal.

FIG. 5A is a schematic view of an exemplary image of a trailer capturedby a rear vehicle camera.

FIG. 6 is a schematic view of an exemplary trailer coupler detector andan exemplary tracker.

FIGS. 7A-7C are schematic side views of a trailer positioned behind thevehicle at different distances therebetween.

FIG. 7D is a schematic top view of a trailer positioned behind thevehicle showing the different positions of a viewport.

FIG. 8 is schematic view of an exemplary arrangement of operations fordetermining the location of the coupler and the center of the bottomportion of the trailer.

FIG. 9 is schematic view of an exemplary arrangement of operationsexecuted by an adaptive tow ball height change detector.

FIG. 10 is schematic view of an exemplary arrangement of operations fordetermining a target and causing a tow vehicle to drive towards thetarget.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

A tow vehicle, such as, but not limited to a car, a crossover, a truck,a van, a sports-utility-vehicle (SUV), and a recreational vehicle (RV)may be configured to tow a trailer. The tow vehicle connects to thetrailer by way of a trailer hitch ball supported by the vehicle and atrailer hitch coupler. It is desirable to have a tow vehicle that iscapable of detecting a tow-bar of the trailer and then detecting andlocalizing a center of the trailer hitch coupler positioned on thetow-bar of the trailer. In addition, it is desirable for the tow vehicleto detect the tow-bar and detect and localize the center of the trailercoupler while the vehicle is moving in a reverse direction towards thetrailer. As such, a tow vehicle with a detection module provides adriver and/or vehicle with information that aids in driving (driver orautonomously) the tow vehicle towards to the trailer in the reversedirection. The detection module is configured to learn how to detect atrailer having a tow-bar and a coupler of trailer and based on thelearned data, the detection module may receive an image from a camera,such as a fisheye camera, having high distortion rate and determine thelocation of the tow-bar and the center of the coupler while the towvehicle approaches the trailer. In addition, a tow vehicle that includesa detection module easily identifies the trailer-tow-bar-couplercombination regardless of the shape and surface the trailer ispositioned on. During a learning phase, the detection module learns todetect any shape tow-bar and any shape coupler, which during a detectionphase, the detection module can use the learned data to accuratelylocalize a center of the coupler and/or key point(s) of the trailer. Insome examples, the learning phase and the detection phase include theuse of a cascade approach on rigidly connected components (e.g., Trailerbody such as Box, V-Nose, Boat etc., trailer frame with two or morewheels for trailer body to be bolted on and coupler tongue sometimes isa part of the frame, sometimes is a separate components belong bolted onthe tow-bar) of the trailer-tow-bar-coupler combination so that acoupler center localization confidence may be improved through therelationships between these components. For example, the cascadeapproach implements an algorithm that detects a trailer body first, thendetects the tow-bar-coupler combination, then zoom in to take a closelook at the coupler center and its surrounding features, and finallydetects the couple tongue. The zoom ratio may be anywhere 1.5-2.5 aslong as there are sufficient features that may be used to identify thecoupler center L_(CCP). The detection module is also capable ofdifferentiating between multiple classes of trailer-tow-bar-couplercombinations parked side by side in one scene, such as in a trailerpark.

Referring to FIGS. 1A-4, in some implementations, a driver of a towvehicle 100 wants to tow a trailer 200 positioned behind the tow vehicle100. The tow vehicle 100 may be configured to receive an indication of adriver selection associated with a selected trailer 200. In someexamples, the driver maneuvers the tow vehicle 100 towards the selectedtrailer 200; while in other examples, the tow vehicle 100 autonomouslydrives towards the selected trailer 200. The tow vehicle 100 may includea drive system 110 that maneuvers the tow vehicle 100 across a roadsurface based on drive commands having x, y, and z components, forexample. As shown, the drive system 110 includes a front right wheel112, 112 a, a front left wheel 112, 112 b, a rear right wheel 112, 112c, and a rear left wheel 112, 112 d. The drive system 110 may includeother wheel configurations as well. The drive system 110 may alsoinclude a brake system 114 that includes brakes associated with eachwheel 112, 112 a-d, and an acceleration system 116 that is configured toadjust a speed and direction of the tow vehicle 100. In addition, thedrive system 110 may include a suspension system 118 that includes tiresassociates with each wheel 112, 112 a-d, tire air, springs, shockabsorbers, and linkages that connect the tow vehicle 100 to its wheels112, 112 a-d and allows relative motion between the tow vehicle 100 andthe wheels 112, 112 a-d. The suspension system 118 may be configured toadjust a height of the tow vehicle 100 allowing a tow vehicle hitch 120(e.g., a tow vehicle hitch ball 122) to align with a trailer hitch 210(e.g., coupler-tow-bar combination 210) supported by a trailer tow-bar214, which allows for connection between the tow vehicle 100 and thetrailer 200.

The tow vehicle 100 may move across the road surface by variouscombinations of movements relative to three mutually perpendicular axesdefined by the tow vehicle 100: a transverse axis X_(v), a fore-aft axisY_(v), and a central vertical axis Z_(v). The transverse axis x extendsbetween a right side and a left side of the tow vehicle 100. A forwarddrive direction along the fore-aft axis Y_(v) is designated as F_(v),also referred to as a forward motion. In addition, an aft or reversedrive direction along the fore-aft direction Y_(v) is designated asR_(v), also referred to as rearward or reverse motion. When thesuspension system 118 adjusts the suspension of the tow vehicle 100, thetow vehicle 100 may tilt about the X_(v) axis and or Y_(v) axis, or movealong the central vertical axis Z_(v).

The tow vehicle 100 may include a user interface 130, such as, adisplay. The user interface 130 receives one or more user commands fromthe driver via one or more input mechanisms or a touch screen display132 and/or displays one or more notifications to the driver. The userinterface 130 is in communication with a vehicle controller 150, whichin turn is in communication with a sensor system 140. In some examples,the user interface 130 displays an image of an environment of the towvehicle 100 leading to one or more commands being received by the userinterface 130 (from the driver) that initiate execution of one or morebehaviors. In some examples, the user display 132 displays one or morerepresentations 136 of trailers 200 positioned behind the tow vehicle100. In this case, the driver selects 134 which representation 136 of atrailer 200 the controller 150 should identify, detect, and localize thecenter of the coupler 212 associated with the selected trailer 200. Inother examples, the controller 150 detects one or more trailers 200 anddetect and localize the center of the trailer coupler 212 of the one ormore trailers. The vehicle controller 150 includes a computing device(or processor) 152 (e.g., central processing unit having one or morecomputing processors) in communication with non-transitory memory 154(e.g., a hard disk, flash memory, random-access memory, memory hardware)capable of storing instructions executable on the computing processor(s)152.

The tow vehicle 100 may include a sensor system 140 to provide reliableand robust driving. The sensor system 140 may include different types ofsensors that may be used separately or with one another to create aperception of the environment of the tow vehicle 100 that is used forthe tow vehicle 100 to drive and aid the driver in make intelligentdecisions based on objects and obstacles detected by the sensor system140 or aids the drive system 110 in autonomously maneuvering the towvehicle 100. The sensor system 140 may include, but is not limited to,radar, sonar, LIDAR (Light Detection and Ranging, which can entailoptical remote sensing that measures properties of scattered light tofind range and/or other information of a distant target), LADAR (LaserDetection and Ranging), ultrasonic sensor(s), etc.

In some implementations, the sensor system 140 includes one or morecameras 142, 142 a-n supported by the vehicle. In some examples, thesensor system 140 includes a rear-view camera 142 a mounted to provide aview of a rear-driving path of the tow vehicle 100. The rear-view camera410 may include a fisheye lens that includes an ultra wide-angle lensthat produces strong visual distortion intended to create a widepanoramic or hemispherical image. Fisheye cameras 142 a capture imageshaving an extremely wide angle of view. Moreover, images captured by thefisheye camera 142 a have a characteristic convex non-rectilinearappearance.

Detection Module 160

The vehicle controller 150 includes a detection module 160 that receivesimages 144 from the rear camera 410 a (i.e., fisheye images) anddetermines the location of the tow-bar 214 and the location of thecoupler 212, for example, a center of the coupler 212 within theimage(s) 144 and in a world coordinate. In some examples, the detectionmodule 160 determines the tow-bar 214 and the center of the coupler 212at a far range, mid-range, and near range distance between the towvehicle 100 and the trailer 200. In some implementations, the detectionmodule 310 includes a training/learning phase 170 followed by adetection phase 180. During the training/learning phase 170, thedetection module 160 executes a trailer and coupler training module 172.In addition, during the detection phase 180, the detection module 160executes the following: ROI proposals determiner 182, trailer andcoupler detector 184, pose and scale estimator 186, a Kalmanmulti-channel correlation filter (MCCF) tracker 188, a confidencecalculation module 190, an adaptive tow ball height change detector 192,and a real-world position estimation module 194.

Training/Learning Phase 170

FIGS. 3 and 4A-4C illustrate the training/learning phase 170. Referringto FIG. 3, The detection module 310 executes a trailer and couplertrainer module 172 for teaching the Kalman MCCF tracker 188 how todetect the coupler 212 and the tow-bar 214. Referring to FIG. 3, thetrainer module 172 receives an input image 144 captured by the rearcamera 410 a, i.e., a fisheye camera. The input image 144 includes aregion of interest (ROI) 400 of either the trailer 200 or thecoupler-tow-bar combination 210 (including the coupler 212 and thetow-bar 214). Referring to FIG. 4A, the input image 144 includes a pixellocation L_(CCP) of the center of the coupler 212 within the image 144and the coupler-tow-bar combination 210 since in this case the trailerand coupler trainer module 172 is learning how to detect the coupler 212within the coupler-tow-bar combination 210 ROI 400 c. While FIGS. 4B and4C show the trailer 200. The pixel location of the center L_(CCP) of thecoupler 212 (FIG. 4A) or the pixel location L_(TCP) of the trailerbottom center at the tow-bar 214 in the training image 144 arepre-determined before the training/learning phase 170, for example,pixel (64, 42) in a (100×100 pixels) training image 144. In someexamples, the ROI 400, 400 c, 400 t is rescaled to a fixed pixel size,for example (100×100 pixels) for the training and learning phase 170.The trainer module 172, for a particular range of pixels, maintains thatthe center L_(CCP) of the coupler 212 or the bottom center L_(TCP) ofthe trailer at the tow-bar 214 in the training image 144 is kept at thesame location within the image 144.

FIG. 4A illustrates a top-down rectified image 146 based on images 144captured by the rearview camera 142 a. The controller 150 adjusts thereceived images 144 to remove distortion caused by the rear camera 142 abeing a fisheye camera. Since the camera 142 a is position above thecoupler-tow-bar combination 210 with respect to the road surface, theimages 144 captured by the camera 142 a of the coupler-tow-barcombination 210 are generally a top-down view of the coupler-tow-barcombination 210. In addition, the trainer module 172 determines a regionof interest (ROI) 400, 400 c within the rectified image 146 thatincludes the coupler-tow-bar combination 210.

FIGS. 4B and 4C are perspective views at different distances of arectified image 146 that includes the trailer 200. FIG. 4B is of arectified image 146 b associated with an image 144 captured at a firstdistance from the trailer 200, while FIG. 4C is of a rectified 146 cassociated with another image 144 captured at a second distance from thetrailer 200, where the second distance is greater than the firstdistance. Since the trailer 200 is positioned behind the tow vehicle100, then the camera 142 a may capture a straight-ahead image 144 of thetrailer 200. Therefore, the rectified images 146 b, 146 c that are basedon the straight-ahead images 144, are also straight-ahead views. In someexamples, the trainer module 172 determines the trailer ROI 400 t withinthe rectified image 146 b, 146 c that include the trailer 200. As shown,the rectified images 146 b, 146 c are also the ROI 400 t identified bythe trainer module 172.

As mentioned above, the trainer module 172 rectifies the captured images144. In some examples, in the top down view shown in FIG. 4A of thecoupler-tow-bar combination 210, the appearance of the tow-bar-coupler(i.e., the aspect ratio of the shape) of the coupler-tow-bar combination210 does not change. The scale change of the top down coupler-tow-barcombination 210 is indicative of the change of height of thecoupler-tow-bar combination 210. In the rear perspective view shown inFIGS. 4B and 4C, the appearance of the trailer 200 does not change givena fixed pitch angle; however, the appearance of the trailer 200 changesif the pitch angle changes. In some examples, the appearance changesdescribed above due to the trailer pitch angle change may be used toestimate trailer pitch angle given a reference image path with a fixedpitch angle. The image path being around the trailer.

Additionally, the appearance changes described above may be used toestimate trailer yaw angle (i.e., orientation) given a reference patchwith a fixed yaw angle. The correlation energy reaches maximum when thetest image matched the trained image. Referring back to FIG. 3, at step302, the trainer module 172 executes a normalizing function on the ROI400, 400 c, 400 t to reduce the lighting variations. Then, at step 304the trainer module 172 executes a gradient function, for example, bydetermining image pixel value difference in both x and y directionwithin the ROI 400, 400 c, 400 t. The trainer module 172 determines amagnitude and an orientation from the following formulas:Magnitude: (sqrt(Gx ² +Gy ²));  (1)Orientation: (arc tan 2(Gy/Gx)).  (2)The trainer module 172 may determine formulas (1) and (2) from thegradient in the x direction (Gx) and the gradient in the y direction(Gy) to determine the directional change in the intensity or color inthe received image ROI 400, 400 c, 400 t and determine a histogram ofthe gradients.

The histogram of gradients (HOG) is a feature descriptor used incomputer vision and image processing to characterize one or more objectswithin an image. The HOG determines a number of bins (for example, 5bins) of gradient orientation in localized portions of the image. Eachbin represents a certain orientation range.

At step 304, based on the HOG, the trainer module 172 determines anumber of bins associated with the HOG. The trainer module 172 executesa cell-wise normalizing function and a block-wise normalizationfunction. During the cell-wise normalization function, the average ofthe gradient magnitude and the gradient orientation over each cell isdetermined, a cell size for example is (5×5 pixels). The trainer module172 determines the number of cells, for example (20×20 pixels), based onthe size of the ROI 400, 400 c, 400 t, (for example 100×100 pixels)divided by a cell size (for example (5×5). If the trainer module 172determines that an average gradient magnitude in a specific cell isbelow zero, then the trainer module 172 set the value of the cell to 0.If the trainer module 172 determines that the average gradientorientation in a specific cell is below zero, then the trainer module172 sets the value of the cell to 0. The trainer module 172 determinesthe average gradient orientation bins based on the average gradientorientations, then multiples the average gradient orientation bins by aninversion of the average gradient magnitude plus 0.1. During theblock-wise normalization function, all cell-wise normalized gradientorientation channels are squared and added up and averaged over cellsize (for example, (5×5)) by the trainer module 172. The trainer module172 sets a sum of gradient square bins (GSsum) to zero when GSsum isless than zero. The trainer module 172 obtains the final sum of gradient(Gsum) by square rooting the GSsum. The trainer module 172 normalizesthe final HOG bins by Gsum, where the cell-normalized gradient bins areagain divided by Gsum plus 0.1. The steps of block 304 accommodate forthe lighting and environmental variations.

At step 306, the trainer module 172 applies a Gaussian filter around atarget location being the center L_(CCP) of the coupler (as ‘pos’ in thefollowing equation) or the pixel location L_(TCP) of the trailer bottomcenter at the tow-bar 214:rsp(i,j)=exp(−((i-pos(1))²+(j-pos(2))²)/(2*S ²)).  (3)S is sigma, a tunable parameter. Since this is a learning step, thetrainer module 172 knows the position of the coupler 212 within theimage, and therefore, applies the Gaussian filter around the knownposition. The Gaussian response, the size of the image patch as ‘rsp’ isalso transformed into frequency domain for fast computation time toobtain Gaussian response in frequency domain as ‘rsp_f’ (being the sizeof the image 144 in frequency domain).

At step 308, the trainer module 172 applies a mask to the ROI 400, 400c, 400 t which improves the performance. The trainer module 172 appliesthe mask to mask out a region surrounding the trailer 200 orcoupler-tow-bar combination 210. At step 310, the trainer module 172applies a cosine window function on the cell/block normalized HoGchannels to reduce the high frequencies of image borders of therectified image 146 (i.e., ROI 400) and transforms the HoG channels intofrequency domain for fast computation time to obtain the HoG_f (HOGfrequency). At step 312, the trainer module 172 calculates Auto- andCross-correlation energies, xxF and xyF respectively. Auto-correlationenergy xxF is obtained by HoG_f multiplied by the transpose of HoG_f.Cross-correlation energy xyF is obtained by rsp_f multiplied by thetranspose of HoG_f. At step 314, the trainer module 172 sums up theAuto-correlation energy xxF and Cross-correlation energy xyF acrossmultiple ROIs 400 in the same range separately. At step 316, the trainermodule 172 solves equation:MCCF=lsqr(xxF+lambda, xyF)  (4)Equation (4) solves for filter banks 322 (i.e., MCCF) and transformsfrom frequency domain back to image domain, at step 318. The filterbanks 322 are multi-channel correlation filter banks 322 that providecharacteristics of the trailer or the coupler-tow-bar combination 210(ie., the trailer hitch 201). As such, the filter banks 322 are laterused to determine the location of a trailer 200 or a coupler 212 withina captured image 144 during a detection phase 180, where the position ofthe trailer 200 or the coupler 212 is not known within the image 144. Atstep 320, the trainer module 172 stores the filter banks 322 (i.e.,MCCF) associated with the trailer 200 and the coupler-tow-barcombination 210 determined by equation (4) in memory hardware 154.

In some implementations, the trainer module 172 separately determinesthree filter banks 322 by executing the steps in FIG. 3 on images 144captured within a far-range, a mid-range and a near-range to accommodatethe appearance, resolution and scale changes of the trailer 200,coupler-tow-bar combination 210 within each image 144.

In some implementations, the training/training and learning phase 170may be executed on a raw fisheye image 144 in a supervised manner, suchas, for example, along the vehicle center line Y with a zero-orientationangle at a specific distance from the trailer 200 in a dynamic drivingscenario. In some examples, the training/training and learning phase 170may be further simplified by using a single image frame or a few imageframes. In some examples, the captured fisheye image 144 is rectified asexplained in FIGS. 4A-4C. The trainer 172 uses the top down view for thefilter learning associated with the coupler-tow-bar combination 210,while the trailer 172 uses the forward perspective view for the filterlearning of the trailer 200. In some examples, the learning images 144are stored in memory 154. The learning images 144 may include thetrailer 200 and tow vehicle in a hitched position where a location ofthe truck tow ball 122 is known. The filter banks 322, whic are learnedduring the training and learning phase 170 based on the rectified images146, may be applied to rectified images 146 for determining a centerlocation L_(CCP) of the trailer 200 and the pixel location L_(TCP) ofthe trailer bottom center at the tow-bar 214 during real-time operation.In some examples, the center location L_(CCP) of the trailer 200 isdetermined by capturing an image 144 in a forward perspective view(FIGS. 4B and 4C). The coupler center may be determined by capturing animage 144 in a top down view in near range (FIG. 4A). In someimplementations, the training and learning phase 170 may also includeimages of the trailer 200 and the tow vehicle 100 in an un-hitchedposition. In some examples, the images 144 of the trailer 200 may becaptured at three to four meters away from the tow vehicle 100. Theimage 144 may include a side of the trailer 200 facing the tow vehicle100.

In some examples, during the training and learning phase 170, thetrailer 200 and coupler-tow-bar combination 210 in the rectified images146 are not in random orientation and have a known orientation angle,such as, for example 0° or 90°. Additionally, the trailer 200 and thecoupler-tow-bar combination 210 are orthogonally connected to oneanother within the images 144 of captured of the forward perspectiveview (FIGS. 4B and 4C) and the top down view (FIG. 4A) similar to theirconnection in real life.

In some implementation, the trainer module 172 rectifies the capturedtop view image 146 a of the tow-bar-coupler ROI 400 c such that thetow-bar-coupler ROI 400 c is at a zero orientation with respect to thetow vehicle 100, i.e., the longitudinal axis Y of the tow vehicle 100while the tow vehicle 100 is hitched to the trailer 200. In addition,the trainer rectifies the perspective image 146 b, 146 c of the trailer200 such that the trailer ROI 400 b is at a zero orientation withrespect to the tow vehicle 100, i.e., the longitudinal axis Y of the towvehicle 100 while the tow vehicle 100 is hitched to the trailer 200.

In some examples, the filter banks 322 that are learned during thelearning process 170 are used to estimate a trailer pitch of the trailer200. The trainer module 172 may mask out regions within the images 144that are not the coupler-tow-bar combination 210 or the trailer 200.Therefore, the trainer module 172 only uses the coupler-tow-barcombination 210 or the trailer 200 regions within an ROI 400 of an image144 and the surrounding regions are masked during training so that thetrained ROI 400 may be used in different environmental conditions withconsistent results. This approach decreases the amount of training datastored in memory 154.

In some implementations, the trainer module 172 analyzes additionalimage key points other than the center L_(CCP) of the coupler 212 or thebottom center L_(TCP) of the trailer at the tow-bar 214. Therefore,during the detection phase 180, the controller 150 can use correlationenergies from the additional key points during real-time operation todetermine a confidence value. Therefore, key points within the trainingROI 400 may be matched with key point identified in images captured inreal time to increase the confidence of the detection phase 180.

In some implementations, the trainer module 172 may generate athree-dimensional (3D) view of the trailer 200, the tow-bar 214 and thecoupler 212 during the training and learning phase 170. Additionally,the trainer module 172 may generate the 3D view in a measuring scale,thus the trainer module 172 can determine a scale between physicaltrailer components and the normalized ROI 400. In some examples, thetrainer module 172 determines and learns shape characteristics of thetrailer 200 and coupler-tow-bar combination 210.

In some examples, the trainer module 172 is executed upon a driver ofthe tow vehicle 100 first using the tow vehicle 100. Therefore, thetrainer module 172 determines the filter banks 322 based on one or moreimages 144 received during the driver's first use of the tow vehicle. Insome examples, the filter banks 322 can also be used in an onlineadaptive learning process or the tow vehicle 100 may receive additionalfilter banks 322 from an online system.

In some examples, the training and learning phase 170 is executed on acloud computing hardware device located separately from the tow vehicle100. For example, the rear camera 142 a of the tow vehicle 100 capturesimages 144 of a rear environment of the vehicle and transmits, via awireless internet connection, the captured images 144 to the cloudcomputing hardware device. The cloud computing hardware device executesthe trainer module 172 and once the filter banks 322 are determined, thecloud computing hardware device transmits the filters banks 322 back tothe tow vehicle 100 by way of the wireless internet connection. The towvehicle 100 receives the filter banks 322 and stores the filter banks322 in memory hardware 154.

Detection Phase 180

Once the training and learning phase 170 is executed and completed, thedetection phase 180 can be executed. The detection module 160 executesthe following: ROI proposals determiner 182, trailer and couplerdetector 184, pose and scale estimator 186, a Kalman multi-channelcorrelation filter (MCCF) tracker 188, a confidence calculation module190, an adaptive tow ball height change detector 192, and a real-worldposition estimation module 194. The ROI proposals determiner 182, whichmay be optionally implements, analyzes the captured images 144 andoutputs an ROI proposal 400 p that includes the coupler-tow-barcombination 210 or the trailer 200. Then the trailer and couplerdetector 184 analyzes the ROI proposals 400 p and uses the ROIs 400 c,400 t from FIG. 4A-4C to locate the trailer (if far) and tow-bar-couplerif close. The pose and scale estimator 186 analyzes the image as a wholeand determines a pose of the trailer 200. Following the Kalman MCCFtracker 188 follows the located trailer or tow-bar coupler (located bythe trailer and coupler detector 184) while the tow vehicle 100 ismoving towards the trailer 200. The confidence calculator 190 calculatesa confidence to determine the position of the coupler. In some examples,the adaptive vehicle tow ball height change detector 192 checks thevehicle tow ball height changes based on the position of the currenttrailer hitch 210. The adaptive vehicle tow ball height change detector192 may be executed while the trailer 200 and the tow vehicle 100 are inthe hitched position, thus, the adaptive vehicle tow ball height changedetector checks the tow-ball height change and verifies that thetrained/learned information which is stored in memory hardware 152matched the features learned from the current tow-ball height so thatpotential collision may be avoided in the next hitching attempt.Finally, the real-world position estimation module 194 determines theposition of the coupler 212 based on the above modules 182-192 in realworld coordinates.

ROI Proposal Determiner 182

FIG. 5 shows a method 500 executed by the ROI proposal determiner 182 todetermine one or more ROI proposals 400 p based on received images 144and automatically detect the coupler-tow-bar combination 210 or thetrailer 200 within the received images 144 by way of applying a tightfit bounding box (ROI) around the coupler-tow-bar combination 210 or thetrailer 200. At block 502, the ROI proposal determiner 182 receives oneor more images 144. At block 504, the ROI proposal determiner 182analyzes the received images 144 and determines a valid area foranalysis. For example, referring to FIG. 5A, for coupler 212 detection,the ROI proposal determiner 182 determines the valid region 530 based oncamera calibration to exclude the area above horizon and the area thatincludes the body of the tow vehicle 100. Similarly, for trailerdetection, the ROI proposal determiner 182 determines the valid region530 based on camera calibration to exclude the area above the trailer200 and the area below the trailer 200.

Following, the ROI proposal determiner 182 may execute one of threemethods to determine the ROI proposal 400 p (i.e., the ROI proposals 400p may include coupler ROI 400 c or trailer ROI 400 t). Method 1: atblock 506, the ROI proposal determiner 182 iteratively applies thelearned filter banks 322 (i.e., MCCF) over a series of scanning windowscovering the image region 530 until a maximum peak is discovered. Thefirst method is known as a brutal force search method and may be usedwhen runtime is not an issue).

Method 2: The ROI proposal determiner 182 may execute method 2 fordetermining ROI proposals 520. At block 508, the ROI proposal determiner182 segments the received region 530 of images 144 using SLIC (SimpleLinear Iterative Clustering) to find superpixels first, then builds aRAG (Region Adjacent Graph) to merge regions with similarities. PixelSimilarity may be based on Intensity, Distance, Color, Edges andTexture. At block 510, the ROI proposal determiner 182 constructs aregion adjacency graph (RAG) based on the segmentation of block 508. TheRAG is a data structure used for segmentation algorithms and providesvertices that represent regions and edges represent connections betweenadjacent regions.

At block, 512 the ROI proposal determiner 182 merges regions within thedetermined region 530 of the image 144. For example, if two adjacentpixels are similar, the ROI proposal determiner 182 merges them into asingle region. If two adjacent regions are collectively similar enough,the ROI proposal determiner 182 merge them likewise. The merged regionis called a super-pixel. This collective similarity is usually based oncomparing the statistics of each region.

At block 514, the ROI proposal determiner 182 identifies possibletrailer super-pixels. A super-pixel qualifies as a trailer region ofinterest if it has minimum and maximum size constraints and excludesirregular shape based on a predetermined trailer type. At block 516, thedetection module 310 identifies and merges possible trailer super-pixelsbased on estimated trailer shape and size at particular image locationsto obtain ROI Proposals 400 p. The ROI proposal determiner 182associates these ROI proposals 400 p with the trained MCCF filters 322to find the peak energy for automatic pixel location of the centerL_(CCP) of the coupler 212 (FIG. 4A) or the pixel location L_(TCP) ofthe trailer bottom center at the tow-bar 214 (FIGS. 4B and 4C).

Method 3: The ROI proposal determiner 182 may execute method 3 fordetermining ROI proposals 520. At block, 518 the ROI proposal determiner182 may use any other object detection and classification methods whichmay generalize the detection of a number of coupler-tow-bar combinations210 or trailers 200, but unable to identify a specific preferred trailerin a trailer park. In some examples, Deep Neural network (DNN) or othergeneralization methods may be used but cannot identify a specifictrailer type. The ROI proposals 520 may be provided by othervision-based object detection and classification methods. The ROIproposals 400 p may also be determined by other sensors such as radar orlidar if available. Multiple ROI proposal methods may be combined toreduce false positive ROI proposals 400 p.

Trailer and Coupler Detector 184

FIG. 6 shows a flow chart of the trailer and coupler detector 184 whilethe tow vehicle 100 approaches the trailer 200. At block 602, the startposition of the tow vehicle 100 is usually within a distance D in metersfrom the trailer 200. At block 604, the trailer and coupler detector 184receives parameters and trained MCCF filters 322 from the memoryhardware 152 (stored by the trained phase). The parameters may include,but are not limited to, as camera intrinsic and extrinsic calibrationinformation, tow vehicle configurations such as a distance between thetow-ball 122 and a front axle of the tow vehicle, wheel base of the towvehicle and tow-ball initial height, and trailer configurations such aswidth of the trailer 200, distance between the coupler 212 and trailerwheel center. Camera intrinsic parameters may include, focal length,image sensor format, and principal point, while camera extrinsicparameters may include the coordinate system transformations from 3Dworld coordinates to 3D camera coordinates, in other words, theextrinsic parameters define the position of the camera center and theheading of the camera in world coordinates. At block 606, the trailerand coupler detector 184 may receive pose estimation of the trailer 200or coupler-tow-bar combination 210 from other modules or a pose andscale estimator 186 may estimate a pose and scale of the trailer 200 orcoupler-tow-bar combination 210 based on the MCCF 322 iteratively todiscover the maximum correlation energy between trained filters and thetrailer and coupler region of interests at specific viewport pose andviewport center.

Then at block 608, the trailer and coupler detector 184 determines adynamic viewport by adjusting the center of the viewport and rotationangle (pitch, yaw, roll) of the viewport, so that the rectified image146 appears to be similar to the trained pattern (i.e., ROI 400 c, 400 tshown in FIGS. 4A-C) that have a known orientation. Referring to FIGS.7A-7D, by changing the viewport center distance from the camera and viewangle such as pitch and yaw, the appearance of the trailer will changeuntil it matches the trained pattern (i.e., ROI 400 c, 400 t shown inFIGS. 4A-C) where the maximum correlation energy is reached. In the realworld, the trailer 200 and tow vehicle 100 are not perfectly alignedwith zero orientation angle between them, and the distance between thetrailer 200 and tow vehicle 100 varies too. The viewports 700 for topdown and forward view are defined with center at (x, y, z) with unit inmm (millimeter) in the world coordinate system. The size of the viewport700 is defined in (width_mm×height_mm), as well as (width_px×height_px)with pitch, yaw and roll rotation angles relative to the autosar(AUTomotive Open System ARchitecture) vehicle coordinate system. Themillimeter per pixel and pixel per millimeter of the image can becalculated. If the rectification viewport 700 center is fixed, the scaleestimation is needed. The rectification viewport center can be adaptivein a way that the trailer size in the rectified image is the same sizeof single channel of the learned filter. Scale estimated can be avoidedby changing the viewport center. If the truck and trailer is notperfectly aligned in real world, the rectification viewport yaw anglecan be adaptive so that the appearance of trailer in forward viewmaintains same orientation and appearance as the training patch. Theviewport yaw angle can be determined iteratively through finding thepeak correlation energy with the trained filter During trailer key pointlocalization, in order to maintain the same size as the single channellearned filter of trailer within rectified image, a series of virtualcameras placed in viewport center in each frame.

The dynamic viewport 700 is configured to adjust the viewing distancewhich is the longitudinal distance of the viewport center from thecamera 142 a and the view angle which is viewport rotation angles (it isalso called viewport pose) such as the pitch, yaw and roll. Therefore,the appearance of the trailer 200 changes based on viewports 700 withdifferent center location and pose configurations. Thus, at block 608,the detector 184 adjusts the viewport of the current image 144 so thatit matches the trained ROI 400. FIG. 7A shows that the viewport is at amiddle distance between the tow vehicle 100 and the trailer 200. FIG. 7Bshows that the viewport is at a closer distance relative to the middledistance. In addition, FIG. 7C shows that the viewport captures the towvehicle 100 and the trailer 200 attached. FIG. 7D shows a top viewillustration of different view center and view angles of the viewport700. (The ‘eye’ location is viewport center, the direction of ‘eye’ isthe viewport pose. The eye location may be indicative, for example, to aperson standing between the camera 142 a and the trailer 200 and lookingat the trailer 200 or looking round.)

At block 610, the trailer and coupler detector 184 determines if it isdetecting a coupler-tow-bar combination 210 or a trailer 200. Therefore,when the trailer and coupler detector 184 determines that it isdetecting a coupler-tow-bar combination 210, then at block 612, thecaptured image 144 includes a top down view of the coupler-tow-barcombination 210 as previously shown in FIG. 4A. However, if the trailerand coupler detector 184 determines that it is detecting a trailer 200,then at block 614, the captures image 144 includes a perspective view ofthe trailer 200 as previously shown in FIGS. 4B and 4C.

Following, at block 616, the trailer and coupler detector 184 instructsthe ROI proposal determiner 182 to determine trailer ROI proposals 400 por tow-bar-coupler ROI proposals 400 p based on the decision of block610. The trailer and coupler detector 184 also generates a lookup table(LUT) for each of the coupler-tow-bar combination 210 and the trailer200 respectively, where the lookup table LUT includes entries from thedynamic viewport (determined at block 608) that correspond to pixellocations in the fisheye image 144.

At block 618, the trailer and coupler detector 184 determines a locationof the peak correlation energy for the coupler center L_(CCP) or thepixel location L_(TCP) of the trailer bottom center at the tow-bar 214the within ROI proposal 400 p of the dynamic viewport image determinedat block 608. Correlation energy refers to a measure of similaritybetween features of two images (such as between ROI proposal 400 p imagepatch and pre-trained trailer patch 400 whose key point is at the bottomcenter of the trailer (L_(TCP)) and also at the peak of gaussian curve,or tow-bar-coupler patch gaussian-weighted at coupler center) as afunction of the pixel location of one relative to another. The peakcorrelation energy in ROI proposal 400 p corresponds to thegaussian-weighted key point in trained image patch 400. In someexamples, the detector 184 executes method 800 shown in FIG. 8 todetermine the peak correlation energy for the coupler center L_(CCP) orthe pixel location L_(TCP) of the trailer bottom center at thecoupler-tow-bar combination 210. At block 802, the detector 184retrieves the ROI proposals 400 p from memory hardware 154 which weredetermined by the determiner 182 and rescaled to the same size as theimage patch used during the training phase. At block 802, the detector184 executes a normalizing function on the ROI proposals 400 p to reducethe lighting variations. At block 804, the detector 184 calculatesmulti-channel normalized HOG features similar to the method executed inFIG. 3 block 304. At block 806, the detector 184 applies cosine windowto HOG channels and transforms them to frequency domain similar to themethod of FIG. 3 block 310. At block 808, the detector 184 determinesthe correlation energy map as a function of pixel location between ROIproposal patch 400 p and trained filter 322 in frequency domain andtransforms back to image domain. At block 810, the detector 184determines the coupler center L_(CCP) or the pixel location L_(TCP) ofthe trailer bottom center at the coupler-tow-bar combination 210 byfinding the pixel location of the peak energy in the correlation energymap.

Referring back to FIG. 6, at block 620, the trailer and coupler detector184 relies on the lookup table to map the coupler center L_(CCP) or thepixel location L_(TCP) of the trailer bottom center at the tow-bar 214determined within the ROI proposal 400 p of the dynamic viewport to theraw fisheye image 144.

Kalman MCCF tracker 188

At block 622, in some examples, the tracker 188 tracks the couplercenter L_(CCP) or the pixel location L_(TCP) of the trailer bottomcenter at the coupler-tow-bar combination 210 within the fisheye image144 based on the mapped coupler center L_(CCP) or the pixel locationL_(TCP) of the trailer bottom center at the tow-bar 214 at block 620.The tracker 188 may include a Kalman MCCF Tracker, which is configuredto track the viewport 700 and key points. At block 622, the tracker 188tracks the viewport center and viewport pose as well as trailer bottomcenter L_(TCP) and coupler center L_(CCP) in raw image. At block 624,the tracker 188 predicts a viewport center and a viewport pose andpredict an updated coupler center L_(CCP) or the pixel location L_(TCP)of the trailer bottom center at the coupler-tow-bar combination 210 inthe new viewport from the predicted key points in raw image. Thepredicted viewport 700 may be used in block 186 as a reference todetermine the next viewport pose and viewport center.

Confidence Calculation 190

In some implementations, the confidence calculation module 190determines a confidence value associated with the tracked coupler centerL_(CCP) or the tracked pixel location L_(TCP) of the trailer bottomcenter at the coupler-tow-bar combination 210. The confidencecalculation module 190 may use the coupler center L_(CCP) or thelocation L_(TCP) of the trailer bottom center at the coupler-tow-barcombination 210 to determine a trailer 200 and a coupler-tow-barcombination 210 orientation in top view or 3D view to check the trailerpose confidence.

Additionally, the confidence calculation module 190 applies a cascadeapproach to determine key point localization confidence. First, theconfidence calculation module 190 detects a trailer body, then theconfidence calculation module 190 detects the coupler-tow-barcombination 210, the confidence calculation module 190 checks theconstraints between two key points. Then the confidence calculationmodule 190 zooms in to take a close look at the coupler center L_(CCP)and its surrounding features. The zoom ratio may be 1.5-2.5 as long asthere are sufficient features that may be used to identify the couplercenter L_(CCP). The confidence calculation module 190 checks correlationenergies and locations of the coupler center L_(CCP) from cascadeapproach. Finally, the confidence calculation module 190 determines ahigh-resolution patch of the coupler itself and the confidencecalculation module 190 analyzes its edge to ascertain the localizationaccuracy of the coupler center. The confidence calculation module 190uses several methods to estimate the one metric to increase theconfidence of localization accuracy. The one metric that the confidencecalculation module 190 is trying to determine is the coupler centerlocation L_(CCP). Therefore, the confidence calculation module 190analyzes the physical constraints and relationships withintrailer-tow-bar-coupler combination, and the key point with varyingsizes of texture and the confidence calculation module 190 also analyzethe high-resolution edge feature of the coupler. Once the confidencecalculation module 190 completes its analysis, the confidencecalculation module 190 determines a confidence of the coupler centerlocation L_(CCP) If confidence is below a threshold, the hitchingoperation may be halted, or the algorithm may be reset to perform awider search. The confidence calculation module 190 may analyze thehistory of the previously determined confidences to determine mismatchbetween trained filters 322 and actual trailer and tow-bar-coupler.

Adaptive Tow Ball Height Change Detector 192

Referring to FIG. 9, in some implementations, after the tow vehicle 100is hitched to the trailer 200, the adaptive tow ball height changedetector 192 determines if the scale (based on one or more capturedimages 144) of features of the attached coupler-tow-bar combination 210are the same as the scale associated with a previously learnedcoupler-tow-bar combination 210.

At block 902, the adaptive tow ball height change detector 192 receivesan image 144 from the rear camera 142 a of the tow vehicle 100 beingattached to the trailer 200. Since the tow vehicle 100 is attached tothe trailer 200, then the coupler center L_(CCP) and the center of thevehicle tow ball 122 overlap. Since the image 144 is associated with thecoupler-tow-bar combination 210, then the image is top-down image asshown in FIG. 4A. At block 904, the adaptive tow ball height changedetector 192 loads learned filter banks 322 and parameters of thetow-bar from memory hardware 154. Following, at block 906, the adaptivetow ball height change detector 192 generates a rectified top-down view146 of the image 144. At block 908, the adaptive tow ball height changedetector 192 finds the coupler center by executing method 800 as shownin FIG. 8. At block 910 the adaptive tow ball height change detector 192crops a number of image patches with different scale factors. The scalefactor may be searched in an iterative manner in the direction where alarger correlation energy is discovered. These image patches are resizedto the size of the patch (ROI 400) used during the training and learningphase 170 and then applied correlation with the trained filters 322. Thescale factor of the image patches with maximum peak correlation energywill be selected. At block 912, the adaptive tow ball height changedetector 192 determines if the scale of the features within the capturedimages 144 with the maximum peak energy is different from the scale offeatures associated with the trained ROI 400 c and if the maximum peakenergy is greater than a predetermined threshold. If one or bothconditions are not met, then the adaptive tow ball height changedetector 192 determines that the image 144 does not include acoupler-tow-bar combination 210 or that the scale of coupler-tow-barcombination 210 has not been changed. However, if both conditions aremet, then at block 914, the adaptive tow ball height change detector 192sets an indication flag to inform the driver that the tow ball heighthas changed, and therefore, the training data may be updated. At block916, the adaptive tow ball height change detector 192 updates the filterbanks 322 (i.e., MCCF) that were learned during the learning phase (FIG.3). At block 918, the adaptive tow ball height change detector 192updates the scale associated with the image patch used during thetraining and learning phase 170 where the training patch has a fixeddimension. At block 920, the adaptive tow ball height change detector192 stored the updated filter banks 322 in memory hardware 154.Therefore, the adaptive tow ball height change detector 192 is aprocedure to synchronize the scale of the tow-bar-coupler patch to thecurrent tow ball height for a safety feature to prevent the tow ball 122and coupler 212 from colliding with each other during hitching. Once itis synchronized, a similar procedure with safety feature may beperformed while the coupler 212 and tow ball 122 are apart within, forexample, one meter from each other, to determine if the coupler 212 ishigher or lower than the tow ball 122 to prevent the coupler 212 and towball 122 from colliding with each other during hitching process. Thisprocess is also referred to as relative height determination.

Real World Position Estimation Module 194

The real-world position estimation module 194 determines the position ofthe trailer coupler 212 determined by the tracker 188 in worldcoordinate system. The viewport center is a three-dimensionalcoordinate. If the same scale is maintained in each image frame duringoperation and the physical width in 3D of the trailer 200 is known, therelationship between 3D coordinates of the trailer 200 and the viewportcenter may be determined. This 3D distance estimation method for trailerrelying on texture width as well as real world trailer width is robustto situations where uneven surfaces is present such as beach, dirt road,grass. Furthermore, the fixed distance between trailer bottom centerL_(TCP) at tow-bar and coupler center in real world is another usefulconstraint to optimize the 3D distance estimation of coupler center andcoupler height.

Therefore, once the real-world position estimation module 194 determinesthe real-world position of the trailer coupler 212, then the real worldposition estimation module 194 sends a drive assist system 196. Based onthe received real world location, the drive assist system 196 determinesa path between the tow vehicle 100 and the trailer 200 leading the towvehicle 100 to align with the trailer 200 for hitching. In addition, thedrive assist system 196 sends the drive system 110 one or more commands198 causing the drive system 110 to autonomously maneuver the towvehicle 100 in a rearwards direction R_(v) towards the trailer 200. Insome examples, the drive assist system 196 instructs the drive system110 to position the tow vehicle 100 such that the fore-aft axis Y_(v) ofthe tow vehicle 100 and the fore-aft axis Y_(T) of the trailer 200 arecoincident.

FIG. 10 provides an example arrangement of operations of a method 1000for determining a location of a target 200, 210, 212, 214 (e.g., thetrailer 200, a coupler-tow-bar combination 210, a coupler 212, and atow-bar 214) positioned behind a tow vehicle 100 using the systemdescribed in FIGS. 1-9. At block 1002, the method 1000 includesreceiving, at data processing hardware 152, images 144 from a camera 142a positioned on a back portion of the tow vehicle 100. The camera 142 amay include a fisheye camera. The images 144 including the target 200,210, 212, 214. At block 1004, the method 1000 includes, applying, by thedata processing hardware 152, one or more filter banks 322 to the images144. The filter banks 322 stored in memory hardware 154 in communicationwith the data processing hardware 152. At block 1006, the method 1000includes determining, by the data processing hardware 152, a region ofinterest (ROI) 400, 400 c, 400 t within each image 144 based on theapplied filter banks 322. The ROI 400, 400 c, 400 t including the target200, 210, 212, 214. At block 1008, the method 1000 includes identifying,by the data processing hardware 152, the target 200, 210, 212, 214within the ROI 400, 400 c, 400 t. At block 1010, the method 1000includes, determining, by the data processing hardware 152, a targetlocation L_(CCP), L_(TCP) of the target 200, 210, 212, 214 including alocation in a real-world coordinate system. At block 1012, the method1000 includes transmitting, from the data processing hardware 152,instructions 195, 198 to a drive system 110 supported by the tow vehicle100 and in communication with the data processing hardware 152. Theinstructions 195, 198 causing the tow vehicle 100 to autonomouslymaneuver towards the location of the target 200, 212, 214 in thereal-world coordinate system.

In some implementations, the method 1000 includes: tracking, by the dataprocessing hardware 152, the target 200, 210, 212, 214 while the towvehicle 100 autonomously maneuvers towards the identified target 200,210, 212, 214; and determining, by the data processing hardware 152, anupdated target location L_(CCP), L_(TCP). The method 1000 may alsoinclude transmitting, from the data processing hardware 152, updatedinstructions 195, 198 to the drive system 110. The updated instructions195, 198 causing the tow vehicle 100 to autonomously maneuver towardsthe updated target location L_(CCP), L_(TCP). In some examples, wherethe camera 142 a is a fisheye camera, the method 1000 further includesrectifying, by the data processing hardware 152, the fisheye images 144before applying the one or more filter banks 322.

In some implementations, the method 1000 includes receiving, at the dataprocessing hardware 152, training images 144 stored in hardware memory154 in communication with the data processing hardware 152. The method1000 may also include determining, by the data processing hardware 152,a training ROI 400, 400 c, 400 t within each received image, thetraining ROI 400, 400 c, 400 t including a target 200, 210, 212, 214.The method 1000 may include determining, by the data processing hardware152, the one or more filter banks 322 within each training ROI 400, 400c, 400 t. In some examples, the method 1000 further includesidentifying, by the data processing hardware 152, a center of thetarget, wherein the target location L_(CCP), L_(TCP) includes a locationof the center of the target.

In some implementations, the target 200, 210, 212, 214 is a coupler 212of a coupler-tow-bar combination 210 supported by a trailer 200.Therefore, the images 144 are a top-down view of the coupler-tow-barcombination 210 as shown in FIG. 4A.

In some implementations, the target 200, 210, 212, 214 is a trailer 200positioned behind the tow vehicle 100 and the target location L_(TCP) isa location of a trailer bottom center at a tow-bar 214. Therefore, theimages 144 are a perspective view of the trailer 200.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium” and“computer-readable medium” refer to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term “machine-readable signal” refers to any signal used to providemachine instructions and/or data to a programmable processor.

Implementations of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Moreover,subject matter described in this specification can be implemented as oneor more computer program products, i.e., one or more modules of computerprogram instructions encoded on a computer readable medium for executionby, or to control the operation of, data processing apparatus. Thecomputer readable medium can be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal, or a combinationof one or more of them. The terms “data processing apparatus”,“computing device” and “computing processor” encompass all apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, or multiple processors or computers.The apparatus can include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them. A propagated signal is an artificially generated signal, e.g.,a machine-generated electrical, optical, or electromagnetic signal thatis generated to encode information for transmission to suitable receiverapparatus.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multi-tasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. Accordingly, otherimplementations are within the scope of the following claims.

What is claimed is:
 1. A method for determining a location of a targetpositioned behind a tow vehicle, the method comprising: receiving, atdata processing hardware, images from a camera positioned on a backportion of the tow vehicle and in communication with the data processinghardware, the images including the target; adjusting, by the dataprocessing hardware, the received images such that an appearance of thetarget within the image matches an orientation of a training targetwithin a training region of interest; applying, by the data processinghardware, one or more filter banks to the adjusted images, the one ormore filter banks including characteristics of the target that arelearned during a learning phase by analyzing one or more trainingimages, each training image including at least one training region ofinterest; determining, by the data processing hardware, a region ofinterest within each image based on the applied filter banks, the regionof interest including the target; identifying, by the data processinghardware, the target within the region of interest; determining, by thedata processing hardware, a target location of the target identifiedwithin the region of interest, the target location including a locationof the target in a real-world coordinate system; and transmitting, fromthe data processing hardware, instructions to a drive system supportedby the vehicle and in communication with the data processing hardware,the instructions causing the tow vehicle to autonomously maneuvertowards the location of the target in the real-world coordinate system.2. The method of claim 1, further comprising: tracking, by the dataprocessing hardware, the target while the tow vehicle autonomouslymaneuvers towards the identified target; determining, by the dataprocessing hardware, an updated target location; and transmitting, fromthe data processing hardware, updated instructions to the drive system,the updated instructions causing the tow vehicle to autonomouslymaneuver towards the updated target location.
 3. The method of claim 1,wherein the camera includes a fisheye camera capturing fisheye images.4. The method of claim 3, further comprising rectifying, by the dataprocessing hardware, the fisheye images before applying the one or morefilter banks.
 5. The method of claim 1, further comprising: receiving,at the data processing hardware, training images stored in hardwarememory in communication with the data processing hardware; determining,by the data processing hardware, a training region of interest withineach received image, the training region of interest including a target;and determining, by the data processing hardware, the one or more filterbanks within each training region of interest.
 6. The method of claim 5,further comprising: identifying, by the data processing hardware, acenter of the target, wherein the target location includes a location ofthe center of the target.
 7. The method of claim 1, wherein the targetis a coupler of a tow-bar-coupler supported by a trailer.
 8. The methodof claim 7, wherein the images are a top-down view of thetow-bar-coupler.
 9. The method of claim 1, wherein the target is atrailer positioned behind the tow vehicle and the target location is alocation of a trailer bottom center at a tow-bar.
 10. The method ofclaim 9, wherein the images are a perspective view of the trailer.
 11. Asystem for determining a location of a target positioned behind a towvehicle, the system comprising: data processing hardware; and memoryhardware in communication with the data processing hardware, the memoryhardware storing instructions that when executed on the data processinghardware cause the data processing hardware to perform operationscomprising: receiving images from a camera positioned on a back portionof the tow vehicle and in communication with the data processinghardware, the images including the target; adjusting the received imagessuch that an appearance of the target within the image matches anorientation of a training target within a training region of interest;applying one or more filter banks to the adjusted images, the one ormore filter banks include characteristics of the target that are learnedduring a learning phase by analyzing one or more training images, eachtraining image including at least one training region of interest;determining a region of interest within each image based on the appliedfilter banks, the region of interest including the target; identifyingthe target within the region of interest; determining a target locationof the target identified within the region of interest, the targetlocation including a location in a real-world coordinate system; andtransmitting instructions to a drive system supported by the vehicle andin communication with the data processing hardware, the instructionscausing the tow vehicle to autonomously maneuver towards the location ofthe target in the real-world coordinate system.
 12. The system of claim11, wherein the operations further comprise: tracking the target whilethe tow vehicle autonomously maneuvers towards the identified target;determining an updated target location; and transmitting updatedinstructions to the drive system, the updated instructions causing thetow vehicle to autonomously maneuver towards the updated targetlocation.
 13. The system of claim 11, wherein the camera includes afisheye camera capturing fisheye images.
 14. The system of claim 13,wherein the operations further comprise rectifying the fisheye imagesbefore applying the one or more filter banks.
 15. The system of claim11, wherein the operations further comprise: receiving training imagesstored in hardware memory in communication with the data processinghardware; determining a training region of interest within each receivedimage, the training region of interest including a target; anddetermining the one or more filter banks within each training region ofinterest.
 16. The system of claim 15, wherein the operations furthercomprise identifying a center of the target, wherein the target locationincludes a location of the center of the target.
 17. The system of claim11, wherein the target is a coupler of a tow-bar-coupler supported by atrailer.
 18. The system of claim 17, wherein the images are a top-downview of the tow-bar-coupler.
 19. The system of claim 11, wherein thetarget is a trailer positioned behind the tow vehicle and the targetlocation is a location of a trailer bottom center at a tow-bar.
 20. Thesystem of claim 19, wherein the images are a perspective view of thetrailer.